Multiple Pathways to Policy Impact: Testing an Uptake Theory with QCA

Policy impact is a complex process influenced by multiple factors. An intermediate step in this process is policy uptake, or the adoption of measures by policymakers that reflect research findings and recommendations. The path to policy uptake often involves activism, lobbying and advocacy work by civil society organisations, so an earlier intermediate step could be termed ‘advocacy uptake’; which would be the use of research findings and recommendations by Civil Society Organisations (CSOs) in their efforts to influence government policy. This CDI Practice Paper by Barbara Befani proposes a ‘broad-brush’ theory of policy uptake (more precisely of ‘advocacy uptake’) and then tests it using two methods: (1) a type of statistical analysis and (2) a variant of Qualitative Comparative Analysis (QCA). The pros and cons of both families of methods are discussed in this paper, which shows that QCA offers the power of generalisation whilst also capturing some of the complexity of middle-range explanation. A limited number of pathways to uptake are identified, which are at the same time moderately sophisticated (considering combinations of causal factors rather than additions) and cover a medium number of cases (40), allowing a moderate degree of generalisation.DFI

Diagnostic evaluation and Bayesian Updating:Practical solutions to common problems

This article discusses several practical issues arising with the application of diagnostic principles to theory-based evaluation (e.g. with Process Tracing and Bayesian Updating). It is structured around three iterative application steps, focusing mostly on the third. While covering different ways evaluators fall victims to confirmation bias and conservatism, the article includes suggestions on which theories can be tested, what kind of empirical material can act as evidence and how to estimate the Bayes formula values/update confidence, including when working with ranges and qualitative confidence descriptors. The article tackles evidence packages (one of the most problematical practical issues), proposing ways to (a) set boundaries of single observations that can be considered independent and handled numerically; (b) handle evidence packages when numerical probability estimates are not available. Some concepts are exemplified using a policy influence process where an institution’s strategy has been influenced by a knowledge product by another organisation

Impact, Innovation and Learning: Towards a Research and Practice Agenda for the Future – Event Report

This report brings together notes and highlights from the International Workshop organised by the Institute of Development Studies (IDS) in Brighton on 26–27 March 2013. The event served as a launch platform for the Centre for Development Impact (CDI), a joint venture between IDS and Itad. The main focus in this workshop report was to share background information about the workshop (participants, programme), as well as its purpose and highlights from the technical discussions. In particular, the latter signposts issues concerning both current practice and policy dilemmas, including areas where further thinking and innovation is needed. The report is written to stimulate thinking and questions for further work, and provides key pointers on an emerging agenda.DFI

Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation

This article proposes a combination of a popular evaluation approach, contribution analysis (CA), with an emerging method for causal inference, process tracing (PT). Both are grounded in generative causality and take a probabilistic approach to the interpretation of evidence. The combined approach is tested on the evaluation of the contribution of a teaching programme to the improvement of school performance of girls, and is shown to be preferable to either CA or PT alone. The proposed procedure shows that established Bayesian principles and PT tests, based on both science and common sense, can be applied to assess the strength of qualitative and quali?quantitative observations and evidence, collected within an overarching CA framework; thus shifting the focus of impact evaluation from ‘assessing impact’ to ‘assessing confidence’ (about impact)

Monitoring and Evaluation Report: Year One

The IDS Accountable Grant is a large and complex four-year grant working across seven Policy Themes, 20 sub-themes, 68 activity domains and over 300 Level 2 Outputs. The scale and diversity of activity under the grant, the dynamic and unpredictable policy environment in which it operates, the number of people and partners involved, together with the imperative to deliver within tight deadlines, highlights the need for a simple yet robust system for tracking planned and completed outputs. This paper reports on the Monitoring and Evaluation of Year 1 of the Accountable Grant, including progress to date, the development of CRM and future developments.DFI

Introduction – Rethinking Impact Evaluation for Development

This IDS Bulletin is the first of two special issues presenting contributions from the event ‘Impact Innovation and Learning: Towards a Research and Practice Agenda for the Future’, organised by IDS in March 2013. The initiative, as well as these two issues, represent a ‘rallying cry’ for impact evaluation to rise to the challenges of a post?MDG/post?2015 development agenda. This introduction articulates first what these challenges are, and then goes on to summarise how the contributors propose to meet these challenges in terms of methodological and institutional innovation. Increasingly ambitious development goals, multiple layers of governance and lines of accountability require adequate causal inference frameworks and less ambitious expectations on the span of direct influence single interventions can achieve, as well as awareness of multiple bias types. Institutions need to be researched and become more impact?oriented and learning?oriented

Introduction – Towards Systemic Approaches to Evaluation and Impact

This IDS Bulletin is the second of two that follow an Institute of Development Studies event seeking to define an agenda for research and practice of development impact evaluation. It focuses on exploring the potential of systems ideas and complexity concepts to meet the increasingly complex challenges of an increasingly ambitious development agenda. In particular, the contributions seek to: (a) redefine ‘learning’ according to the number of ‘learning loops’ involved; (b) understand how to identify the most relevant impact evaluation questions; (c) simulate systems states in two sectors (leather and health) following the implementation of (combinations of) different policy options and other events; and finally, (d) shake the foundations of the impact evaluation institutional system, recommending that the notions of multiple perspectives and system boundaries are fully embraced, and that the system ultimately transitions from an ‘evaluation industrial complex’ to an ‘evaluation adaptive complex’. While the issue is a step in the right direction, much more work remains to be done

Testing the Waters: A Qualitative Comparative Analysis of the Factors Affecting Success in Rendering Water Services Sustainable Based on ICT Reporting

This research conducted by WaterAid, Itad and IRC aims to understand the factors that facilitate and inhibit the success of ICT-based reporting to improve rural water supply sustainability. They compare eight ICT initiatives in Kenya, Bolivia, Timor Leste, Uganda, Tanzania and India, to identify the most crucial factors that lead to success or failure

Addressing and Mitigating Violence: Uptake Strategy

The overarching purpose of the Addressing and Mitigating Violence theme (AMV) is to generate useful analysis to tackle policy dilemmas relating to ‘newer’ forms of violence and organised crime. An important goal of work across the AMV theme will be to increase the capabilities of partners and stakeholders in each of the contexts where we work to identify more effective ways of policy influence. The strategy is a working document that reflects uptake as an ongoing process which is responsive to emerging policy opportunities, learning and partnership activities. As the AMV programme progresses through years two to four, the strategy will be continually updated and strengthened through incorporating new approaches to knowledge uptake and dissemination based on ongoing learning.DFI

Contribution Analysis and Estimating the Size of Effects: Can We Reconcile the Possible with the Impossible?

While contribution analysis provides a step-by-step approach to verify whether and why an intervention is a contributory factor to development impact, most contribution analysis studies do not quantify the ‘share of contribution’ that can be attributed to a particular support intervention. Commissioners of evaluations, however, often want to understand the size or importance of a contribution, not least for accountability purposes. The easy (and not necessarily incorrect) response to this question would be to say that it is impossible to do so. However, in this CDI Practice Paper written by Giel Ton, John Mayne, Thomas Delahais, Jonny Morell, Barbara Befani, Marina Apgar and Peter O’Flynn, we explore how contribution analysis can be stretched so that it can give some sense of the importance of a contribution in a quantitative manner. The first part of the paper introduces the approach of contribution analysis and presents ideas to capture the change process in theories of change and system maps. The second part presents research design elements that include ranking or quantitative measures of impact in the verification of the theory of change and resulting contribution story